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ABSTRACT 

Optimal control of systems governed by delayed- 
differential equations is explored by using the control 
theory developed for systems governed by ordinary differential 
equations. A simple algorithm for producing a suboptimal 
control law with restricted feedback is presented. Two 
examples illustrate the computational method. 
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I. INTRODUCTION 



A. DELAYED-DIFFERENTIAL EQUATIONS 

In many physical systems, such as rocket engines [1,2], 
antirolling stabilization systems for destroyers [3] and 
pipeline recycling systems for chemical reactors [4], the 
most accurate mathematical representation of system 
dynamics is found to be a matrix system of differential 
equations with finite time delay in the arguments of some of 
the state variables. 

x(t) = Ax(t) + Bx(t-x) + Cu(t) (1) 

t _> 0 , x •> 0 

x(t ) = 4>(t ) ; -t _< t <_ o (2) 

Equations of this form have been known as differential- 
difference, hysterodif f erential , functional differential, 
differential with transport lag, differential with deviating 
arguments, and delayed-diff erential equations, to name just 
a few. For the purposes of this research, equations of the 
form of (1) are referred to as delayed-dif ferential 
equations . 

B. THE OPTIMAL REGULATOR PROBLEM 

The research to be described is concerned with a solution 
of the optimal regulator problem. Simply stated, this 
problem seeks the control function u*(t) (a member of the 
set of admissible controls u(t)) which minimizes a quadratic 
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cost function of the form 



J(<f>(t) ,x(t) ,u(t) ) 



/ [x’(t)Qx(t) + u' (t )Ru(t ) ]dt > 0 (3) 

0 



subject to the constraints of equations (1) and (2). Q is a 

2 2 
n positive semidefinite matrix; R is a Z positive definite 

matrix; i <_ n . 

C. NOTATION 

Notation follows that established by Bryson and Ho [5]. 
In general, it is as follows: 

1. Column vectors are denoted by lower case letters. 
Single subscripted lower case letters are elements of 
vectors . 

2. Matrices are indicated by upper case letters. Doubly 
subscripted lower case letters represent elements of 
a matrix. 

3. Matrix transpose is indicated by the prime symbol (’). 

The following are deviations from the general rules 

established above. 

1. t represents the scalar independent variable time. 

2. t denotes the scalar time delay, possibly a function 
of time and state variables. 

3. J is a scalar function of matrices. 

4. The lower case letters i,j,£,m, and n are used as 
indexing variables. 

5. The total derivative is indicated by d. 

6. A damped natural frequency is represented by uij . 



D . PROBLEM STATEMENT 
1. Introduction 

Inclusion of the delay term t in the state variable 
equation greatly complicates solution of the optimal 
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regulator problem. The well established theory of ordinary 
differential equation systems can not be applied with 
equation (1). In addition, although the necessary theory 
and computational procedures have been developed [6] for 
computation of the optimal control law, implementation of 
that lav; is often found to be either impractical or undesir- 
able. Implementation of the optimal control law requires 
feedback of a continuum of states, x(£), t - x £ £ <_ t , 
which are often not readily available from the system being 

i 

controlled. 

2 . Purpose 

The purpose of this research is to develop a method • 
for approximating the optimal control law for delayed- 
differential systems by utilizing the optimal control 
formulation for ordinary differential systems. In addition, 
a simple algorithm is offered for restricted feedback cases. 

E. BACKGROUND 
1. General 

Anderson and Soudack [7] indicate that the study of 
the delayed-dif ferential equation appears to have started 
with John Bernoulli in 1728, but most presently available 
information has been published since 19*10 Bibliographies 
published by R. Weiss [8] in 1959 and Chosky [9] in I960 
contain references to more than 350 separate papers dealing 
with the delayed-dif ferential equation and associated 
problems. The annotated bibliography by R. V/eiss surveys 
English language papers published between 1935 and 1958. 
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The work by Chosky is supplemental to the bibliography of 
Weiss and covers the period of 1916 to 1959, including 
reference to numerous foreign language papers. 

Prior to I960, several papers were published which 
addressed the problem of approximating the delay term so 
that the desired control law could be obtained. 

In a paper published in 19*10, Mason and Philbrick [10] 

devised a liquid level analog to a thermal process with time 

lag. This model was used to examine response to various 

/ 

types of control systems. A related work by Ziegler and 
Nichols [11] considered various control settings for auto- 
matic control circuits with process lag. The optimal control 
was determined from the area under a state time-history 
curve. Bretoi [12] and Minorsky [3] used first order differ- 
ential equations to approximate control systems with time 
lags. After the control lav; was determined from the 
approximation, the actual system was adjusted to give the 
desired results. 

In an attempt to explain why, under certain circum- 
stances, a pendulum began to spontaneously oscillate with a 
frequency higher than its own damped natural frequency or 
that of the externally applied forcing function, Minorsky [13] 
expressed the system model as 

x 1 (t) + a 1 x 1 (t) + b £ (t-t) + c 1 x 1 (t) = 0 (4) 

Considering the time delay term t to be small, x^Ct-t) was 
expanded in a Taylor series to yield 
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(-l) n x[ n+1) (t) + ... - TX 1 (t) + x 1 (t) 

For very large n and small x this expression converges to a 
term of the form e a . Replacing the delayed term In equation 
(M) by e ° the following system model was obtained: 

• • 

x 1 (t) = (a x - b 1 e -a ) x^ ( t ) + c 1 x 1 (t) = 0 

This equation was studied to determine regions of stability. 

The major work in the theory of delayed-dif f erential 
equations has been undertaken since I960. This increasing 
emphasis may in part be attributed to the rising application 
of such equations to automatic control systems as well as 
the development of high speed computers. 

2 . Development of Theory 

Bellman [14] and El’sgol'ts [15] discussed the 
existence and uniqueness of solutions to equations of the 
form of (1) . 

Krosovskii [16], appears to have been one of the 
earliest developers of an optimal control formulation for 
the linear quadratic problem with time delays in the states. 
Although he gave no explicit definitions for the parameters 
required to actually determine the optimal control law; the 
feedback was in the form of a linear combination of the 
states. Extension of Pontryagin's maximum principle to 
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systems with multiple delays was accomplished by Kharatishvii 
[17,18]. These works have served as a basis for numerous 
other papers on differential-difference equations. Orguz- 
toreli [19], Chyung and Lee [20] have also worked with the 
derivation of necessary conditions for optimality in the 
form of the maximum principle. 

Ross and Flugge-Lotz [6] developed an optimal linear 
control law, similar in form to Krosovskii ' s , for a system 
in which the time delay had been normalized to one. (In a 
later work, Ross [21] showed that a system with any number 
of delay terms can be transformed into a system with one 
delay which can be normalized to one.) This paper also 
presented the sufficient conditions for the existence of 
admissible control functions and sufficient conditions for 
a linear control law to be optimal. Kushner and Barnea [22] 
showed that a linear control law for (1) subject to a qua- 
dratic cost function in the form of (3) was optimal with 
respect to the class of square integrable control laws, if 
the solution to (1) was continuous. 

3 . Approximation of Delayed-Dif ferential Equations 

Because of the complexity of solving the optimal 
regulator problem when the system dynamics involve delayed- 
dif ferential equations, considerable effort has been 
directed toward approximating the delayed-dif ferential 
equation by systems of ordinary differential equations. 

Repin [ 23 ] has shown that equation (1) can be 
approximated to any desired accuracy by a series of 
differential equations. 
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For systems in which the delay term is small, Jen- 

A 

Wei [24] developed a procedure for replacing the delay term 
by a series of piece-wise linear functions which lead to 
a system of linear state variable equations with no time 
delay. Hess [25] has extended this procedure to a system 
in which the delay was not restricted. 

Results similar to those obtained by Ross and Flugge- 
Lotz [6] were developed by Soliman and Ray [26,27] by 
approximating the delay with an arbitrarily large set of 
differential equations. Similar procedures have been used 
by Westdal and Lehn [28] for a related problem. 
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II. THEORY OP THE OPTIMAL CONTROL OP 
DELAYED-DIFFERENTIAL SYSTEMS 



A. EXISTENCE AND UNIQUENESS 

For an equation of the following form, 

x(t ) = f(t,x(t),x(t — t ) ) 
x(t) = ( t ) , t q — x £ t <_ t Q 

T(t ,x(t ) ) _> 0 

/ 

El'sgol'ts [15] has shown that if the functions f, <f) , x are 
continuous, the solution exists. If in addition 



3f 




3f 




d<Kt) 


, and 


dx 


3 (x(t ) ) 




3 ( x(t - x ) ) 




dt 


dt 



are bounded in the vicinity of the initial value (a Lipschitz 
condition in x(t)), then the solution is also unique. 

Since it has been shown [6] that the optimal control, 
u*(t), is of the form 

i 0 

u*(t) = -R 1 C'[K 0 x(t) + / K 1 (0)x(t + 0)d6] 

-T 

the solution to the optimal regulator problem (equations 1-3) 
exists and is unique if the above Lipschitz condition is 



satisfied. 



B. CONTROLLABILITY 



L. Weiss [29] developed algebraic sufficient conditions 
for delayed-diff erential equations with time varying coeffi- 
cients to be controllable. These conditions reduce to the 
ordinary algebraic criteria for controllability as the delay 
becomes negligible. The paper presents a method for computing 
the columns for the controllability matrix. 

Considering a system with constant coefficients, Hewer 
[ 30 ] has shown that if the system in which t ->0 is control- 

i 

lable, then the delayed equation is controllable for any 
r _> 0. The converse of this statement is not always true. 

For the time invariant case, the results obtained by Weiss 
can be reduced to those obtained by Hewer. 

C. DEVELOPMENT OF THE OPTIMAL CONTROL LAW 

Ross and Flugge-Lotz [6] and Ross [21] have developed 
the optimal control law for equation (1) subject to (3) as 
follows ; 

1. Admissible Control 

If (1) is controllable, the set of admissible control 
functions, u(t), are those measureable functions in L o [0,°°] 
which yield a finite value for (3) for any set of initial 
conditions 4>(t). 

2 . Sufficient Conditions for a Linear Feedback 

Control Law to be Optimal 

The optimal control law of (1) subject to the cost 
function (3) and a specified set of initial condition 
functions (2) is of the form 
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u*(t ) = -R 1 C , [K 0 x(t) + / K 1 (0)x(t + 0)d0] (5) 

-T 

t _> 0 ; x(t)=<f>(t) > -x _< t <_ 0 



a. u*(t) is a stable control law (admissible in this case) 

b. Kq (a symmetric positive definite matrix), K 1 ( 0 ) and 
K 2 (0,O satisfy the following relationships 

t 

A'K 0 + KqA - K 0 CR _1 K 0 + K 1 '(0) + 1^(0) + Q = 0 (6) 



dK, (0) . 

= (A’ - K q CR ■ l C')K 1 (0) + K 2 (O,0) -t <0 <0 (7) 

3K 5 (0,O 3K„ ( 0,0 ■, 

TT + W = -VtOCR-V^Ce) (8) 

— T < ^ < 0 , — T < 0 < 0 



K 1 (-x) = K q B (9) 

K 2 (-t,6) = B , K 1 (0) (10) 



If B = 0, equivalent to a system with no delay term, the 
above conditions reduce to the familiar algebraic Riccati 
equation. The first term in u*(t) is the optimal control 
law for the undelayed system; the second and more complicated 
term accounts for the addition of the delay term. 
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Under the above conditions, the quadratic cost function 
(3) can be expressed in terms of the initial condition 
function (2). 

0 

J(<J>,u*) = <j>’ (0)K Q <f>(0) + 2<J> ' ( 0 ) / K 1 (e)«K6)d0 

-T 

0 0 

+ / / 4 >» (c)K 2 (e,c)«i>(e)dcde (11) 

-T -T 

Although the control lav;, u*(t), is optimal for an arbitrary 

t 

initial condition function which is continuously differenti- 
able, the value of (3) is, as would be expected, a direct 
function of the initial conditions. 

3. Stability of the Optimal Control Lav; 

In order to apply the optimal control lav;, some 
approximate method of computation is required. Once the 
control law is calculated, there is no known algorithmic 
method by which the stability of the control can be 
assessed [6]. Determination of stability requires that the 
closed loop system performance be investigated for a set of 
initial condition functions large enough to cover those 
expected to be encountered by the control system. If no 
unstable behavior is detected, the control law can be 
assumed to be stable in that particular application. 



17 



III. APPROXIMATION OF THE OPTIMAL CONTROL LAW 
FOR DELAYED- DIFFERENTIAL SYSTEMS 



A. APPROXIMATION OF DELAYED-DIFFERENTIAL EQUATIONS 

Repin [23] has shown that equation (1) can be approxi- 
mated by a series of equations, determined by defining a new 
set of state variables. The state variables are defined as 
follows : 



x x *(t) 
x 2 * (t ) 



,x(t ) 



x(t 



T 

m-1 



) 



x 1 3f (t) = x(t 



ix 

m-1 



) 



( 12 ) 



x m *(t) = x(t - t) 

Where m is a positive integer equal to or greater than two, 
x ± *(t) is a n*l column vector composed of the elements of 
the original state vector. 

Using simple forward difference formulas for estimating 
derivatives, the following set of equations can be developed: 
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x 1 *(t ) 



= A x,*(t) +Bx *(t) + C u(t ) 

jl m 



x 2 *(t) 



= (x 1 ^(t) - x 2 *(t)) 



X ± *(t) 



= (x i _*(t) - x ± «(t)) 



( 13 ) 



V^> = (x m _i(t) - x m *(t)) 2=i 

f 

This series of first order, linear, matrix differential 
equations can be shown to be expressible as 



y(t ) = a q y(t ) + c Q u(t ) 



( 1*0 



where 



A o - 



A 

(m-1) 



0 

0 



0 

- (m-1) 
x 

(m-1) 



-(m-1) 



B 



I ... 0 



-(m-1) 



(15) 



Qq s 



Q 0 



0 0 



0 0 



(16) 
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y(t ) 



(17) 





Repin has shown that' by selecting m large enough, 
equation (1) can be approximated to any degree of accuracy 
desired, and in the limit as m -> °° , the representation 
presented above becomes exact. 

B. OPTIMAL AND SUBOPTIMAL CONTROL OF 
ORDINARY DIFFERENTIAL SYSTEMS 

For the time invariant linear system of differential 
equations 

x(t) = Aq x(t) + Cq u(t) ; t _> 0 ; x(0) = <f> 

with a quadratic cost function in the form of (3), the opti- 
mal control law has been shown to be 



where V is the symmetric positive definite solution to the 
algebraic Riccati equation 



u«(t) = -R 1 C 0 ’ v x ( t ) 



( 18 ) 



VA 0 + A 0 * V - VC Q R" 1 C 0 ’ V + Q 0 - 0 



(19) 



20 



When the optimal control law is applied, the value of 
the cost function (3) can be expressed as 

J = cf>' V cp 

Elkind and Falb [31] have shown that if some control 
u(t) = -Gx(t) (G is a matrix of constants conformal to x(t)) 
is applied, instead of the optimal control law ( 1 8 ) , the cost 
function can be expressed as 

J Q = 4-' V 0 <}) > 4»' (20) 

further, if the eigenvalues of (Aq - Cq G) have negative real 
parts, Vq is the symmetric positive definite solution to 

(A 0 - C Q G)'Vq + Vq(Aq - Cq G) + G» R" 1 G + Qq = 0 (21) 

These expressions can be used to determine the value of 
(3) for any stable feedback composed of a linear combination 
of states. 

C. APPROXIMATION OF THE CONTROL LAW 

It has been shown that (1) can be approximated by a 
system of differential equations 

. y(t) = A 0 y(t) + C 0 u(t) ; t > 0 (22) 

y (t ) = <F ; t = o 

with y (t ) , y (t ) , A Q , and C Q as defined by (12-17). The 
initial condition matrix is 
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<f>(0) 



4> 



<f>( 



-T 

m-1 



) 



(23) 



<K-t) 



The cost function is 

oo 

J(y(0) ,y(t) ,u(t) ) = / [y'(t) Q n y(t) + u'(t) Ru(t)] dt 

0 u 
t 

(24) 

The hypothesis here is that an accurate approximation to 
the optimal control lav; (5) for the delayed-dif ferential 
system (1) can be obtained from the optimal control lav: ( 18 ) 
for an ordinary differential system. This ordinary differen- 
tial system is, of course (22), the one which approximates 
the delayed-dif ferential system (1). 

Rigorous proof of this hypothesis is difficult. Instead, 
the following section shows that this approximate method is 
exactly equivalent to the computational approximation which 
Ross utilizes in solving for his optimal control law for 
system ( 1 ) . 

D. EQUIVALENCE OF COMPUTATIONAL APPROACHES* 

1 . Ross's Method 

In order to compute the optimal control law (5)> 

Ross [21] replaces the conditional equations (6-10) by 



*For convenience, and without loss of generality, the 
time delay t has been normalized to unity in the discussion 
which follows. 
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